Use of comparative sensor data to determine orientation of head relative to body

ABSTRACT

Methods and systems are described that involve a wearable computing device or an associated device determining the orientation of a person&#39;s head relative to their body. To do so, example methods and systems may compare sensor data from the wearable computing device to corresponding sensor data from a tracking device that is expected to move in a manner that follows the wearer&#39;s body, such a mobile phone that is located in the wearable computing device&#39;s wearer&#39;s pocket.

CROSS-REFERENCE TO RELATED APPLICATION

This patent application is a continuation of U.S. application Ser. No.13/631,454, which was filed on Sep. 28, 2012, the contents of which areentirely incorporated herein by reference as if fully set forth in thisapplication.

BACKGROUND

Unless otherwise indicated herein, the materials described in thissection are not prior art to the claims in this application and are notadmitted to be prior art by inclusion in this section.

Computing devices such as personal computers, laptop computers, tabletcomputers, cellular phones, and countless types of Internet-capabledevices are increasingly prevalent in numerous aspects of modern life.Over time, the manner in which these devices are providing informationto users is becoming more intelligent, more efficient, more intuitive,and/or less obtrusive.

The trend toward miniaturization of computing hardware, peripherals, aswell as of sensors, detectors, and image and audio processors, amongother technologies, has helped open up a field sometimes referred to as“wearable computing.” In the area of image and visual processing andproduction, in particular, it has become possible to consider wearabledisplays that place a very small image display element close enough to awearer's (or user's) eye(s) such that the displayed image fills ornearly fills the field of view, and appears as a normal sized image,such as might be displayed on a traditional image display device. Therelevant technology may be referred to as “near-eye displays.”

Near-eye displays are fundamental components of wearable displays, alsosometimes called “head-mounted displays” (HMDs). A head-mounted displayplaces a graphic display or displays close to one or both eyes of awearer. To generate the images on a display, a computer processingsystem may be used. Such displays may occupy a wearer's entire field ofview, or only occupy part of wearer's field of view. Further,head-mounted displays may be as small as a pair of glasses or as largeas a helmet.

Emerging and anticipated uses of wearable displays include applicationsin which users interact in real time with an augmented or virtualreality. Such applications can be mission-critical or safety-critical,such as in a public safety or aviation setting. The applications canalso be recreational, such as interactive gaming.

SUMMARY

In one aspect, an exemplary computer-implemented method may involve acomputing device: (i) detecting sensor data that is indicative of anassociation between movement of a tracking device and body movement;(ii) in response to detecting the sensor data that is indicative of thepositional association: (a) determining a forward-backward body axis ofa body and (b) determining a base orientation of a tracking devicerelative to the forward-backward body axis; (iii) determining a firstorientation of a head-mountable device (HMD) relative to the trackingdevice; and (iv) determining a first head orientation relative to thebody based on both: (a) the first orientation of the HMD relative to thetracking device and (b) the base orientation of the tracking devicerelative to the forward-backward body axis.

In another aspect, a non-transitory computer readable medium may havestored therein instructions that are executable to cause a computingsystem to perform functions comprising: (i) detecting sensor data thatis indicative of an association between movement of a tracking deviceand body movement; (ii) in response to detecting the sensor data that isindicative of the positional association: (a) determining aforward-backward body axis of a body and (b) determining a baseorientation of a tracking device relative to the forward-backward bodyaxis; (iii) determining a first orientation of a head-mountable device(HMD) relative to the tracking device; and (iv) determining a first headorientation relative to the body based on both: (a) the firstorientation of the HMD relative to the tracking device and (b) the baseorientation of the tracking device relative to the forward-backward bodyaxis.

In a further aspect, a computing system may include a non-transitorycomputer readable medium and program instructions stored on thenon-transitory computer readable medium. The program instructions may beexecutable by at least one processor to: (i) detect sensor data that isindicative of an association between movement of a tracking device andbody movement; (ii) in response to detecting the sensor data that isindicative of the positional association: (a) determine aforward-backward body axis of a body and (b) determine a baseorientation of a tracking device relative to the forward-backward bodyaxis; (iii) determine a first orientation of a head-mountable device(HMD) relative to the tracking device; and (iv) determine a first headorientation relative to the body based on both: (a) the firstorientation of the HMD relative to the tracking device and (b) the baseorientation of the tracking device relative to the forward-backward bodyaxis.

These as well as other aspects, advantages, and alternatives, willbecome apparent to those of ordinary skill in the art by reading thefollowing detailed description, with reference where appropriate to theaccompanying drawings.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1A illustrates a wearable computing system according to anexemplary embodiment.

FIG. 1B illustrates an alternate view of the wearable computing deviceillustrated in FIG. 1A.

FIG. 1C illustrates another wearable computing system according to anexemplary embodiment.

FIG. 1D illustrates another wearable computing system according to anexemplary embodiment.

FIG. 2 is a simplified illustration of a network via which one or moredevices may engage in communications, according to an exemplaryembodiment.

FIG. 3 is a flow chart illustrating a method, according to an exampleembodiment.

FIG. 4A is a top-down illustration of a scenario in which an HMD wearerhas a tracking device on their person.

FIG. 4B is a top-down illustration of a scenario in which an HMD wearerhas a tracking device on their person while walking.

FIG. 5 is a flow chart illustrating a method for determining theorientation of an HMD relative to a tracking device.

FIG. 6 illustrates a side-view of a scenario in which an HMD wearermoves their head with respect to their body.

DETAILED DESCRIPTION

Exemplary methods and systems are described herein. It should beunderstood that the word “exemplary” is used herein to mean “serving asan example, instance, or illustration.” Any embodiment or featuredescribed herein as “exemplary” is not necessarily to be construed aspreferred or advantageous over other embodiments or features. Theexemplary embodiments described herein are not meant to be limiting. Itwill be readily understood that certain aspects of the disclosed systemsand methods can be arranged and combined in a wide variety of differentconfigurations, all of which are contemplated herein.

I. Overview

There are scenarios where it may be useful for a head-mountable display(HMD), such as a glasses-style wearable computer, to know how a wearer'shead is oriented with respect to their body. For example, the positionof the head relative to the body may be used to provideaugmented-reality style graphics in the HMD, in which certainapplications or graphics may then be rendered at the same place relativeto the body. As a specific example, a wearer might look over their leftshoulder to view an e-mail application, and look over their rightshoulder to view a web browser. As such, the e-mail application and webbrowser may appear to exist in space over the user's left and rightshoulder, respectively.

An example embodiment may utilize data from an HMD and a tracking devicelocated on the body of the HMD wearer to determine the orientation ofthe head relative to the wearer's body. The tracking device may takevarious forms, such as computing device the wearer typically has ontheir person (e.g., a mobile phone) or a separate dedicated trackingdevice that the user can put in their pocket or attach to theirclothing. Importantly, example embodiments may help an HMD to determinehead position relative to a wearer's body, without requiring that theuser mount the tracking device at a certain location and in a certainposition on the body.

In an exemplary embodiment, an HMD (or a remote computing system incommunication with an HMD) may compare the HMD's sensor data to thesensor data from a tracking device that is expected or inferred to havea certain physical association to the wearer's body. For example, theHMD may determine that the wearer has a mobile phone in their pocket (orin another location where the movement of the phone is generallyexpected to follow the movement of the wearer's body). The HMD may thendetect when the wearer is walking and determine the forward direction bysensing which direction the body is moving while walking. The HMD ormobile phone can then use sensor data from the mobile phone to determinethe orientation of the phone with respect to the body (e.g., theorientation with respect to an axis aligned with direction the wearer iswalking). In addition, the HMD may use magnetometer data (and possiblygyroscope data) from both the HMD and the mobile phone to determine eachdevice's orientation relative to magnetic north. The HMD can then usethis information to determine the orientation of the HMD with respect tothe tracking device. Then, to determine the orientation of the HMD withrespect to the body, the HMD may offset its orientation with respect tothe tracking device by the orientation of the tracking device withrespect to the body.

The above technique, which can provide the horizontal rotation of thehead relative to the body, relies on the difference between magnetometerreadings at the HMD and a tracking device such as a mobile phone. In afurther aspect, the differences between the accelerometer and/orgyroscope readings of the tracking device and the HMD can be used in asimilar manner to determine the pitch and/or yaw of the head relative tothe body. Therefore, by analyzing the differences in sensor data betweenthe accelerometers, magnetometers, and/or gyroscopes of an HMD and atracking device, the HMD may detect three-dimensional changes in headposition relative to the body.

II. Illustrative Systems

Systems and devices in which exemplary embodiments may be implementedwill now be described in greater detail. In general, an exemplary systemmay be implemented in or may take the form of a wearable computer. Inparticular, an exemplary system may be implemented in association withor take the form of a head-mountable display (HMD), or a computingsystem that receives data from an HMD, such as a cloud-based serversystem.

However, an exemplary system may also be implemented in or take the formof other devices, such as a mobile phone, among others. Further, anexemplary system may take the form of non-transitory computer readablemedium, which has program instructions stored thereon that areexecutable by at a processor to provide the functionality describedherein. An exemplary system may also take the form of a device such as awearable computer or mobile phone, or a subsystem of such a device,which includes such a non-transitory computer readable medium havingsuch program instructions stored thereon.

FIG. 1A illustrates a wearable computing system according to anexemplary embodiment. In FIG. 1A, the wearable computing system takesthe form of a head-mounted device (HMD) 102 (which may also be referredto as a head-mountable display). It should be understood, however, thatexemplary systems and devices may take the form of or be implementedwithin or in association with other types of devices, without departingfrom the scope of the invention. As illustrated in FIG. 1A, thehead-mounted device 102 comprises frame elements including lens-frames104, 106 and a center frame support 108, lens elements 110, 112, andextending side-arms 114, 116. The center frame support 108 and theextending side-arms 114, 116 are configured to secure the head-mounteddevice 102 to a user's face via a user's nose and ears, respectively.

Each of the frame elements 104, 106, and 108 and the extending side-arms114, 116 may be formed of a solid structure of plastic and/or metal, ormay be formed of a hollow structure of similar material so as to allowwiring and component interconnects to be internally routed through thehead-mounted device 102. Other materials may be possible as well.

One or more of each of the lens elements 110, 112 may be formed of anymaterial that can suitably display a projected image or graphic. Each ofthe lens elements 110, 112 may also be sufficiently transparent to allowa user to see through the lens element. Combining these two features ofthe lens elements may facilitate an augmented reality or heads-updisplay where the projected image or graphic is superimposed over areal-world view as perceived by the user through the lens elements.

The extending side-arms 114, 116 may each be projections that extendaway from the lens-frames 104, 106, respectively, and may be positionedbehind a user's ears to secure the head-mounted device 102 to the user.The extending side-arms 114, 116 may further secure the head-mounteddevice 102 to the user by extending around a rear portion of the user'shead. Additionally or alternatively, for example, the HMD 102 mayconnect to or be affixed within a head-mounted helmet structure. Otherpossibilities exist as well.

The HMD 102 may also include an on-board computing system 118, a videocamera 120, a sensor 122, and a finger-operable touch pad 124. Theon-board computing system 118 is shown to be positioned on the extendingside-arm 114 of the head-mounted device 102; however, the on-boardcomputing system 118 may be provided on other parts of the head-mounteddevice 102 or may be positioned remote from the head-mounted device 102(e.g., the on-board computing system 118 could be wire- orwirelessly-connected to the head-mounted device 102). The on-boardcomputing system 118 may include a processor and memory, for example.The on-board computing system 118 may be configured to receive andanalyze data from the video camera 120 and the finger-operable touch pad124 (and possibly from other sensory devices, user interfaces, or both)and generate images for output by the lens elements 110 and 112.

The video camera 120 is shown positioned on the extending side-arm 114of the head-mounted device 102; however, the video camera 120 may beprovided on other parts of the head-mounted device 102. The video camera120 may be configured to capture images at various resolutions or atdifferent frame rates. Many video cameras with a small form-factor, suchas those used in cell phones or webcams, for example, may beincorporated into an example of the HMD 102.

Further, although FIG. 1A illustrates one video camera 120, more videocameras may be used, and each may be configured to capture the sameview, or to capture different views. For example, the video camera 120may be forward facing to capture at least a portion of the real-worldview perceived by the user. This forward facing image captured by thevideo camera 120 may then be used to generate an augmented reality wherecomputer generated images appear to interact with the real-world viewperceived by the user.

The sensor 122 is shown on the extending side-arm 116 of thehead-mounted device 102; however, the sensor 122 may be positioned onother parts of the head-mounted device 102. The sensor 122 may includeone or more of a gyroscope or an accelerometer, for example. Othersensing devices may be included within, or in addition to, the sensor122 or other sensing functions may be performed by the sensor 122.

The finger-operable touch pad 124 is shown on the extending side-arm 114of the head-mounted device 102. However, the finger-operable touch pad124 may be positioned on other parts of the head-mounted device 102.Also, more than one finger-operable touch pad may be present on thehead-mounted device 102. The finger-operable touch pad 124 may be usedby a user to input commands. The finger-operable touch pad 124 may senseat least one of a position and a movement of a finger via capacitivesensing, resistance sensing, or a surface acoustic wave process, amongother possibilities. The finger-operable touch pad 124 may be capable ofsensing finger movement in a direction parallel or planar to the padsurface, in a direction normal to the pad surface, or both, and may alsobe capable of sensing a level of pressure applied to the pad surface.The finger-operable touch pad 124 may be formed of one or moretranslucent or transparent insulating layers and one or more translucentor transparent conducting layers. Edges of the finger-operable touch pad124 may be formed to have a raised, indented, or roughened surface, soas to provide tactile feedback to a user when the user's finger reachesthe edge, or other area, of the finger-operable touch pad 124. If morethan one finger-operable touch pad is present, each finger-operabletouch pad may be operated independently, and may provide a differentfunction.

FIG. 1B illustrates an alternate view of the wearable computing deviceillustrated in FIG. 1A. As shown in FIG. 1B, the lens elements 110, 112may act as display elements. The head-mounted device 102 may include afirst projector 128 coupled to an inside surface of the extendingside-arm 116 and configured to project a display 130 onto an insidesurface of the lens element 112. Additionally or alternatively, a secondprojector 132 may be coupled to an inside surface of the extendingside-arm 114 and configured to project a display 134 onto an insidesurface of the lens element 110.

The lens elements 110, 112 may act as a combiner in a light projectionsystem and may include a coating that reflects the light projected ontothem from the projectors 128, 132. In some embodiments, a reflectivecoating may not be used (e.g., when the projectors 128, 132 are scanninglaser devices).

In alternative embodiments, other types of display elements may also beused. For example, the lens elements 110, 112 themselves may include: atransparent or semi-transparent matrix display, such as anelectroluminescent display or a liquid crystal display, one or morewaveguides for delivering an image to the user's eyes, or other opticalelements capable of delivering an in focus near-to-eye image to theuser. A corresponding display driver may be disposed within the frameelements 104, 106 for driving such a matrix display. Alternatively oradditionally, a laser or LED source and scanning system could be used todraw a raster display directly onto the retina of one or more of theuser's eyes. Other possibilities exist as well.

FIG. 1C illustrates another wearable computing system according to anexemplary embodiment, which takes the form of an HMD 152. The HMD 152may include frame elements and side-arms such as those described withrespect to FIGS. 1A and 1B. The HMD 152 may additionally include anon-board computing system 154 and a video camera 206, such as thosedescribed with respect to FIGS. 1A and 1B. The video camera 206 is shownmounted on a frame of the HMD 152. However, the video camera 206 may bemounted at other positions as well.

As shown in FIG. 1C, the HMD 152 may include a single display 158 whichmay be coupled to the device. The display 158 may be formed on one ofthe lens elements of the HMD 152, such as a lens element described withrespect to FIGS. 1A and 1B, and may be configured to overlaycomputer-generated graphics in the user's view of the physical world.The display 158 is shown to be provided in a center of a lens of the HMD152, however, the display 158 may be provided in other positions. Thedisplay 158 is controllable via the computing system 154 that is coupledto the display 158 via an optical waveguide 160.

FIG. 1D illustrates another wearable computing system according to anexemplary embodiment, which takes the form of an HMD 172. The HMD 172may include side-arms 173, a center frame support 174, and a bridgeportion with nosepiece 175. In the example shown in FIG. 1D, the centerframe support 174 connects the side-arms 173. The HMD 172 does notinclude lens-frames containing lens elements. The HMD 172 mayadditionally include an on-board computing system 176 and a video camera178, such as those described with respect to FIGS. 1A and 1B.

The HMD 172 may include a single lens element 180 that may be coupled toone of the side-arms 173 or the center frame support 174. The lenselement 180 may include a display such as the display described withreference to FIGS. 1A and 1B, and may be configured to overlaycomputer-generated graphics upon the user's view of the physical world.In one example, the single lens element 180 may be coupled to the innerside (i.e., the side exposed to a portion of a user's head when worn bythe user) of the extending side-arm 173. The single lens element 180 maybe positioned in front of or proximate to a user's eye when the HMD 172is worn by a user. For example, the single lens element 180 may bepositioned below the center frame support 174, as shown in FIG. 1D.

FIG. 2 illustrates a schematic drawing of a computing device accordingto an exemplary embodiment. In system 200, a device 210 communicatesusing a communication link 220 (e.g., a wired or wireless connection) toa remote device 230. The device 210 may be any type of device that canreceive data and display information corresponding to or associated withthe data. For example, the device 210 may be a heads-up display system,such as the head-mounted devices 102, 152, or 172 described withreference to FIGS. 1A-1D.

Thus, the device 210 may include a display system 212 comprising aprocessor 214 and a display 216. The display 210 may be, for example, anoptical see-through display, an optical see-around display, or a videosee-through display. The processor 214 may receive data from the remotedevice 230, and configure the data for display on the display 216. Theprocessor 214 may be any type of processor, such as a micro-processor ora digital signal processor, for example.

The device 210 may further include on-board data storage, such as memory218 coupled to the processor 214. The memory 218 may store software thatcan be accessed and executed by the processor 214, for example.

The remote device 230 may be any type of computing device or transmitterincluding a laptop computer, a mobile telephone, or tablet computingdevice, etc., that is configured to transmit data to the device 210. Theremote device 230 and the device 210 may contain hardware to enable thecommunication link 220, such as processors, transmitters, receivers,antennas, etc.

In an illustrative embodiment, a remote device 230 such as a mobilephone, tablet computing device, a laptop computer, etc., could beutilized as a tracking device. More specifically, device 210 may be anHMD, and sensor data from sensors on the remote device 230, such datafrom one or more magnetometers, accelerometers, and/or gyroscopes, maybe compared to corresponding sensor data from the HMD to determine theposition of the HMD wearer's head relative to their body.

The remote device 230 could also be a remote computing system that isconfigured to perform functions on behalf of device 210; i.e., a “cloud”computing system. In such an embodiment, the remote computing system mayreceive data from device 210 via link 220, perform certain processingfunctions on behalf of device 210, and then send the resulting data backto device 210.

Further, device 210 may be in communication with a number of remotedevices, such as remote device 230. For example, an HMD could be incommunication with a remote computing system that provides certainfunctionality to the HMD, as well as a mobile phone, or another suchdevice that may serve as a tracking device in embodiments describedherein. Other examples are also possible.

In FIG. 2, the communication link 220 is illustrated as a wirelessconnection; however, wired connections may also be used. For example,the communication link 220 may be a wired serial bus such as a universalserial bus or a parallel bus. A wired connection may be a proprietaryconnection as well. The communication link 220 may also be a wirelessconnection using, e.g., Bluetooth® radio technology, communicationprotocols described in IEEE 802.11 (including any IEEE 802.11revisions), Cellular technology (such as GSM, CDMA, UMTS, EV-DO, WiMAX,or LTE), or Zigbee® technology, among other possibilities. The remotedevice 230 may be accessible via the Internet and may include acomputing cluster associated with a particular web service (e.g.,social-networking, photo sharing, address book, etc.).

III. Illustrative Methods

FIG. 3 is a flow chart illustrating a method 300, according to anexample embodiment. Illustrative methods, such as method 300, may becarried out in whole or in part by an HMD, such as the head-mountabledisplays shown in FIGS. 1A to 1D. Example methods, or portions thereof,could also be carried out by a tracking device (e.g., a mobile phone oranother mobile device that the wearer of an HMD might carry on theirperson), alone or in combination with an HMD. Further, an examplemethod, or portions thereof, may be carried out by computing device thatis in communication with an HMD and/or in communication with a trackingdevice. An example method may also be carried out by other types ofcomputing devices and/or combinations of computing devices, withoutdeparting from the scope of the invention.

As shown by block 302, method 300 involves a computing device detectingthat sensor data is indicative of an association between movement of atracking device and body movement. For example, an HMD may detect thatdata from an accelerometer and/or gyroscope of the HMD (and/or data fromsuch sensors on the tracking device) corresponds to the HMD wearerwalking forward.

In response to detecting the sensor data that is indicative of such anassociation, the computing device may perform a calibration routine todetermine an orientation of the tracking device with respect to the HMDwearer's body. (This assumes that the HMD is being worn; if it is not,then the calibration routine may still be implemented, but will producean orientation of the tracking device with respect to a hypotheticalposition of the body.) More specifically, the computing device maydetermine a forward-backward body axis of a body, as shown by block 304.Further, the computing device may determine a base orientation of atracking device relative to the forward-backward body axis, which may bereferred to as θ_(TD-B), as shown by block 306.a

Once the computing device has performed the calibration routine, thecomputing device may use the base orientation θ_(TD-B) of the trackingdevice relative to the forward-backward body axis in order to determinean orientation of the head relative to the body. For instance, in theillustrated method 300, the computing device may determine a firstorientation θ_(HMD-TD) _(_) ₁ of the HMD relative to the trackingdevice, as shown by block 308. The computing device may then determine afirst head orientation θ_(H-B) _(_) ₁ relative to the body (e.g.,forward-backward body axis) based on both: (a) the first orientationθ_(HMD-TD) _(_) ₁ of the HMD relative to the tracking device and (b) thebase orientation θ_(TD-B) of the tracking device relative to theforward-backward body axis, as shown by block 310. In a further aspect,the computing device may initiate a computing action based on the firsthead orientation θ_(H-B) _(_) ₁ relative to the body, as shown by block312.

Method 300 may be described herein with reference to the scenarioillustrated in FIG. 4A. In particular, FIG. 4A is a top-downillustration of a scenario 400 in which an HMD wearer has a trackingdevice on their person. More specifically, FIG. 4A shows a top down viewof the head 402 and the body 404 of a person that is wearing an HMD 406and has a mobile phone 408 on their person. FIG. 4A also shows thenorth-south (N-S) and east-west (E-W) axes that are defined by magneticnorth (N).

A. Detecting an Association Between Movement of a Tracking Device andBody Movement

Referring to FIG. 3, at block 302, a computing device may use varioustechniques to detect an association between the movement of a trackingdevice and body movement. Further, to do so, various types of sensordata may be utilized.

For example, an HMD or a mobile phone may receive data that isindicative of movement of the HMD and/or data that is indicative ofmovement of the tracking device, such as data from a gyroscope,accelerometer, and/or a compass on one or both of the devices. Thecomputing device may then analyze such sensor data and determine that itis characteristic of movement along the forward-backward body axis.

For instance, when the wearer of an HMD is walking or driving, and has amobile phone in their pocket (which serves as the tracking device), theaccelerometer(s) and/or gyroscopes of the mobile phone and/or of the HMDmay indicate movement of the mobile phone and/or movement of the HMDthat is characteristic of walking. Similarly, there may be movementpatterns that are indicative of the wearer driving or riding in a car.More generally, the computing device may detect the association when itdetects other an action where the wearer's body is typically facing inthe direction they are moving, such that the movement of the wearer'sbody has significant directional component in the forward or backwarddirection (e.g., along the forward-backward body axis Y_(B) of thewearer's body, as shown in FIG. 4A).

Techniques for analyzing sensor data, such as data from accelerometers,gyroscopes, and/or compasses, to detect actions where the wearer's bodyis typically aligned with the forward-backward body axis Y_(B) of thewearer's body, such as walking, driving or riding in a car, are known inthe art. Accordingly, the details of such techniques or not discussedfurther herein.

When the wearer is engaged in an action such as walking or driving, itmay often be the case that the wearer has a tracking device such as amobile phone on their person (e.g., in their pocket, purse, backpack,etc.) or nearby in an orientation that is relatively stable with respectto their body (e.g., sitting on the passenger seat of their car whilethe HMD wearer is driving). In such a scenario, the position of thetracking device may therefore provide an indication of the position ofthe wearer's body. For example, when an HMD wearer has a mobile phone oranother type of tracking device in their pocket, movements of the mobilephone will typically follow the movements of the wearer's body.

Accordingly, an HMD may use its orientation with respect to the mobilephone to determine the HMD's orientation with respect to the wearer'sbody. And since the HMD may generally be assumed to align with thewearer's head (possibly after adjusting to account for translationbetween the wearer's field of view and sensors on the HMD), the HMD mayuse the HMD's orientation with respect to the wearer's body as a measureof the orientation of the wearer's head with respect to the wearer'sbody.

B. Calibration

i. Determining the Forward-Backward Body Axis

At block 304, computing device may use various techniques to define theforward-backward axis Y_(B). In particular, and as noted above, awearer's movement that typically has a significant forward or backwardcomponent along the forward-backward body axis Y_(B), such as walking,may be interpreted to indicate the association between movement of atracking device and movement of the body. Therefore, when a computingdevice detects sensor data that is indicative of such a movement by thewearer, some or all of this data may be analyzed to determine thedirection of forward body movement, which may then be used to define theforward-backward body axis Y_(B).

As a specific example, FIG. 4B is a top-down illustration of a scenario450 in which an HMD wearer has a tracking device on their person whilewalking. More specifically, FIG. 4B shows a top-down view of the head452 and the body 454 of a person that is wearing an HMD 456 and has amobile phone 458 on their person (e.g., in their pocket). As is oftenthe case, the person's head 452 and body 454 are facing forward, in thedirection 460 that the person is walking.

To determine the forward-backward body axis Y_(B), a computing devicesuch as the mobile phone 458 and/or HMD 456 may first determine anup-down body axis. To do so, the mobile phone 458 and/or HMD 456 maydetermine the direction of gravitational force, which is aligned withthe up-down body axis (assuming the wearer is in an upright position).In particular, the mobile phone 458 and/or HMD 456 may utilize data fromthe HMD's accelerometer(s) and/or gyroscopes(s), and/or data from thetracking device's accelerometer(s) and/or gyroscopes(s). Then, todetermine the forward-backward axis Y_(B), the computing device may thenevaluate accelerometer readings that are perpendicular to the downwarddirection (i.e., perpendicular to the direction of gravitational force).

More specifically, when the movement that indicates the associationbetween the mobile phone 458 and the wearer's body 454, theaccelerometer data from the mobile phone 458 and/or from the HMD 456 maybe expected to have the highest variance in the forward direction. Assuch, the HMD 456 and/or the mobile phone 458 may analyze theaccelerometer data to determine the forward direction of the body bydetermining the direction having the highest variance, or possibly thedirection having the highest average magnitude, over a predeterminedperiod of time (e.g., a two-second window). The computing device maythen align the forward-backward body axis Y_(B) with the direction thatis determined to be forward.

ii. Determining the Offset Between the Tracking Device and Body

Once the computing device has determined the wearer's theforward-backward body axis Y_(B), the computing device may determine thebase orientation θ_(T-B) of the tracking device relative to theforward-backward body axis. Since the association between the movementof the tracking device and body movement has been detected, it may beinferred that the tracking device will follow the wearer's body. Assuch, the base orientation θ_(TD-B) of the tracking device relative tothe forward-backward body axis may be used as a reference to determinethe orientation of the HMD (and thus the head) with respect to thewearer's body.

For instance, if the tracking device is a mobile phone that is locatedin the wearer's pocket, it may be expected that the tracking device islikely to stay in the wearer's pocket for at least a short period oftime. The wearer's pocket may further be expected to hold the mobilephone in substantially the same place with respect to the wearer's body.Thus, the orientation θ_(TD-B) of the mobile phone 408 with respect tothe wearer's body 406 may be expected and/or assumed to remainsubstantially the same over a certain period of time. Therefore, at agiven point in time, the orientation θ_(TD-B) of the mobile phone 408with respect to the wearer's body 406 may be used to offset theorientation of the HMD relative to the tracking device, to determine theorientation of the HMD relative to the body.

More specifically, at block 306 of method 300, the determination of thebase orientation θ_(TD-B) of the tracking device relative to theforward-backward body axis may involve the computing device determiningan angle between a forward direction of the tracking device and thedirectional component along the forward-backward body axis. To do so, acompass and/or other sensors of the tracking device may be configured soas to indicate the orientation of the tracking device relative tomagnetic north. Data from the compass and/or the other sensors maytherefore be used to determine the direction that the tracking device isfacing.

As a specific example, in FIG. 4A, the direction that the mobile phone408 is facing may define the tracking device's forward-backward axisY_(TD), as shown in FIG. 4. The computing device may then use theforward direction along the tracking device's forward-backward axisY_(TD) and the forward direction along the body's forward-backwardY_(B), to determine the base orientation θ_(TD-B) of the tracking devicewith respect to the body.

Note that in FIG. 4A, the tracking device's forward-backward axis Y_(TD)is shifted such that it aligns with the up-down axis of the wearer'sbody, instead of the up-down axis of the body. This is done forillustrative purposes. More specifically, in an illustrative embodiment,the orientation θ_(TD-B) of the tracking device relative to the body ismeasured in a parallel plane to the plane of the forward-backward bodyaxis Y_(B), (e.g., parallel to the yaw planes of the wearer's body andthe wearer's head). Therefore, in such an embodiment, the shift of thetracking device's forward-backward axis Y_(TD) does not effect on howthe orientation θ_(TD-B) is calculated.

C. Determining the Orientation of the HMD Relative to the TrackingDevice

Referring again to example method 300 of FIG. 3, various techniques maybe used to determine the first orientation θ_(HMD-TD) _(_) ₁ of the HMDrelative to the tracking device. For example, FIG. 5 is a flow chartillustrating a method for determining the orientation of an HMD relativeto a tracking device. In particular, method 500 may determine theorientation of an HMD relative to a tracking device based on: (a)magnetometer data associated with the HMD and (b) magnetometer dataassociated with the tracking device.

More specifically, at block 502 of method 500, a computing device maydetermine a first orientation θ_(HMD-N) _(_) ₁ of the HMD relative tomagnetic north. This determination may be based on magnetometer dataassociated with the HMD (e.g., data captured by the HMD's magnetometer).The computing device may also determine a first orientation θ_(TD-N)_(_) ₁ of the tracking device relative to magnetic north, as shown byblock 504. These determinations may be based on magnetometer dataassociated with the HMD (e.g., data captured by the HMD's magnetometer),and on magnetometer data associated with the tracking device (e.g., datacaptured by the tracking device's magnetometer), respectively. Thecomputing device may then determine the first orientation θ_(HMD-TD)_(_) ₁ of the HMD relative to the tracking device based on a differencebetween (a) the first orientation θ_(HMD-N) _(_) ₁ of the HMD relativeto magnetic north and (b) the first orientation θ_(TD-N) _(_) ₁ of thetracking device relative to magnetic north, as shown by block 506.

For instance, FIG. 4A shows the orientation θ_(HMD-TD) of the HMD 406relative to the mobile phone 408 in example scenario 400. Applyingmethod 500 in scenario 400, block 502 may involve the HMD 406 (or aremote computing system in communication with the HMD) analyzing datafrom a compass (e.g., a magnetometer) and/or other sensors attached toor integrated in the HMD 406, and determining the first orientation ofthe HMD 406 relative to magnetic north (θ_(HMD-N) _(_) ₁) therefrom.Similarly, block 504 may involve the HMD 406 and/or the mobile phone 408analyzing data from the mobile phone's compass and/or other sensors ofthe mobile phone 408, and determining the first orientation of themobile phone 408 relative to magnetic north (θ_(TD-N) _(_) ₁) therefrom.The HMD 406 may then calculate its orientation relative to the mobilephone 408 (θ_(HMD-TD) _(_) ₁) as being equal to the angular differencebetween θ_(HMD-N) _(_) ₁ and θ_(TD-N) _(_) ₁.

D. Determining the First Head Orientation Relative to the Body

At block 310 of method 300, various techniques may be used to determinethe head orientation relative to the wearer's body.

In particular, by determining the base orientation of a tracking devicerelative to the forward-backward body axis (θ_(TD-B)) at block 306, thecomputing device has a quantitative measure of how the HMD is positionedwith respect to the tracking device. Further, by determining theorientation of the HMD relative to the tracking device θ_(HMD-TD) atblock 308, the computing device has a quantitative measure of how theHMD is positioned with respect to the tracking device. As such, acomputing device may determine the orientation θ_(HMD-TD) of the HMDrelative to the tracking device, and then adjust according to the baseorientation θ_(TD-B) to determine the orientation θ_(H-B) of thewearer's head relative to their body. As a specific example, andreferring again to FIG. 4, a computing device may offset θ_(HMD-TD) toaccount for the orientation θ_(TD-B) of the mobile phone 408 withrespect to the body 404, in order to determine the orientation θ_(H-B)of the head 402 with respect to the body 404.

Further, it may be assumed that the base orientation θ_(TD-B) of thetracking device relative to the forward-backward body axis stayssubstantially the same over certain periods of time (such as when thetracking device is a mobile phone in the wearer's pocket or purse). Itis also possible that the computing device may explicitly determine thatthe tracking device has not or is unlikely to have moved relative to thebody since calculating θ_(TD-B). In either case, theinitially-determined base orientation θ_(TD-B) of the tracking devicerelative to the forward-backward body axis can thus be used to offset asubsequent calculation of θ_(HMD-TD) and determine the orientation ofthe head relative to the body (θ_(H-B)) at the time of the subsequentcalculation.

E. Determining Three-Dimensional Head Position Relative to Body in

In the above description for FIGS. 3 to 5, differences between themagnetometer readings from an HMD and a mobile phone or another trackingdevice are used to determine two-dimensional orientation of the HMDwearer's head with respect to the wearer's body (i.e., the yaw of thehead with respect to the body). In some embodiments, additional sensordata, may be used to determine the orientation of the wearer's headrelative to their body in three dimensions. For example, the differencesbetween the accelerometer readings of the tracking device and the HMDcan be used in a similar manner to determine upward and downwardmovements of the head relative to the body.

For example, FIG. 6 illustrates a side-view of a scenario 600 in whichan HMD wearer moves their head with respect to their body. Inparticular, the person moves their head from a first position 602 a to asecond position 602 b. In the first position 602 a, the person is facingforward, such that there their head is substantially aligned with theirbody 604. In other words, there is no rotation or pitch of the head withrespect to their body, so the axes of the body are generally parallel tothe axes of the head. However, when the person moves their head to thesecond position 603 b, the position of their head is such that there isa pitch Φ_(H-B) of their head 602 relative to their body 604. Further,roll of the head may be determined by combining the analysis ofdifferences in accelerometer data with analysis of differences inmagnetometer data (and possibly differences in gyroscope data as well)between the HMD and the tracking device.

More specifically, to determine the pitch Φ_(H-B) of the head withrespect to the ground, a low-pass filter may be applied to theaccelerometer signal from the tracking device and/or the HMD tosubstantially cancel out other motions and determine a direction ofgravity; i.e., to determine the downward direction and thus provide thealignment of the UP-DOWN axis shown in FIG. 6. The HMD could thendetermine the pitch Φ_(H-B) of the head based on measured gravity vector(e.g., in the downward direction on the UP-DOWN axis in the coordinateframe of the HMD. (Note that the coordinate frame of the HMD is shown byY_(HMD) and Z_(HMD) in FIG. 6.) The angle of the gravity vector ascompared to the Y_(HMD) axis may then be used to determine the pitchΦ_(H-B) of the head.

Further, to determine the roll of the head with respect to the ground,an HMD may apply a similar technique as that used to determine thepitch, except that the HMD (or associated computing device) maydetermine the downward direction and the direction to the right or leftof the body. In particular, the HMD may determine a coordinate framedefined by the direction of gravity (e.g., the UP-DOWN axis shown inFIG. 6), and the right-left axis of the body (e.g., the X_(B) axis shownin FIG. 4B). The angle of the gravity vector as compared to the X_(HMD)axis (as shown in FIG. 4A) may then be used to determine the roll of thehead.

By performing the above the HMD may determine the pitch and/or roll ofthe HMD, and thus the angle of the head, with respect to the ground(i.e., with respect to gravity). The same process may be carried out bya tracking device such as a phone to determine the pitch and/or roll ofthe tracking device with respect to the ground (i.e., with respect togravity). Then, the pitch and/or roll of the HMD with respect to thebody may be determined by using the axis defined by gravity (e.g., theUP-DOWN axis) in a similar manner as the forward-backward axis is usedto determine the yaw.

F. Re-calibration

As noted above, it may be assumed that the base orientation θ_(TD-B) ofthe tracking device relative to the forward-backward body axis stayssubstantially the same over certain periods of time. However, it ispossible that θ_(TD-B) can change over time. For instance, theorientation of a mobile phone with respect to the body may change when,e.g., the mobile phone shifts within the wearer's pocket or the wearermoves the mobile phone from a console in their car to their purse.Accordingly, an example method may further involve a computing devicere-calibrating to compensate for changes in a tracking device's positionrelative to the body.

For example, in some embodiments, a computing device may periodicallyrepeat blocks 302 to 306 of method 300 in order to re-determine the baseorientation θ_(TD-B) of the tracking device relative to theforward-backward body axis. By doing so, the computing device may updatethe offset that is applied to the orientation θ_(HMD-TD) of the HMDrelative to the tracking device.

In some embodiments, a computing device may additionally oralternatively monitor or periodically check whether the orientation ofthe tracking relative to the body has changed. The computing device maythen re-calibrate when it determines that the orientation of trackingdevice relative to the body has changed (or is likely to have changed).For example, a computing device may receive or detect an indication thatthe tracking device has moved in relation to the body (e.g., in amessage from the tracking device). In response to the indication thatthe tracking device has moved in relation to the body, the computingdevice may monitor the HMD's sensor data and/or the tracking device'ssensor data until it detects sensor data that indicates a calibrationevent (e.g., the wearer walking forward), and then re-determine the baseorientation θ_(TD-B) of the tracking device relative to theforward-backward body axis.

IV. Conclusion

While various aspects and embodiments have been disclosed herein, otheraspects and embodiments will be apparent to those skilled in the art.The various aspects and embodiments disclosed herein are for purposes ofillustration and are not intended to be limiting, with the true scopeand spirit being indicated by the following claims.

We claim:
 1. A computer-implemented method comprising: detecting, by acomputing device, sensor data that is indicative of an associationbetween movement of a tracking device and body movement; in response todetecting the sensor data that is indicative of the association:determining a forward-backward body axis of a body corresponding to awearable computing device; and determining a base orientation of atracking device relative to the forward-backward body axis (θ_(TD-B));determining a first orientation of the wearable computing devicerelative to the tracking device (θ_(HMD-TD) _(_) ₁); and determining afirst head orientation relative to the body (θ_(H-B) _(_) ₁) based onboth: the base orientation (θ_(TD-B)) and the first orientation(θ_(HMD-TD) _(_) ₁), wherein determining the first head orientationrelative to the body (θ_(H-B) _(_) ₁) comprises offsetting the firstorientation (θ_(HMD-TD) _(_) ₁) by the base orientation of the trackingdevice relative to the forward-backward body axis (θ_(TD-B)).
 2. Themethod of claim 1, wherein the computing device is the wearablecomputing device.
 3. The method of claim 1, wherein the tracking deviceis a mobile phone.
 4. The method of claim 1, wherein detecting thesensor data that is indicative of the association between movement ofthe tracking device and body movement comprises: receiving sensor datafrom at least one of the wearable computing device and the trackingdevice, wherein the sensor data is indicative of movement; anddetermining that the sensor data is characteristic of movement along theforward-backward body axis.
 5. The method of claim 4, whereindetermining that the sensor data is characteristic of movement along theforward-backward body axis comprises determining that the sensor data ischaracteristic of walking.
 6. The method of claim 4, wherein the sensordata comprises at least one of: (a) gyroscope data associated with thewearable computing device, (b) accelerometer data associated with thewearable computing device, (c) gyroscope data associated with thetracking device, and (d) accelerometer data associated with the trackingdevice.
 7. The method of claim 1, wherein determining a forward-backwardbody axis of a body comprises determining a direction of forward bodymovement.
 8. The method of claim 1, wherein determining a baseorientation of the tracking device relative to the forward-backward bodyaxis comprises determining an angle between a forward-facing directionof the tracking device and the directional component along theforward-backward body axis.
 9. The method of claim 1, wherein the firstorientation of the wearable computing device relative to the trackingdevice is determined based on both: (a) magnetometer data associatedwith the wearable computing device and (b) magnetometer data associatedwith the tracking device.
 10. The method of claim 9, wherein determiningthe first orientation of the wearable computing device relative to thetracking device comprises: determining a first orientation of thewearable computing device relative to magnetic north based on themagnetometer data associated with the wearable computing device;determining a first orientation of the tracking device relative tomagnetic north based on the magnetometer data associated with thetracking device; and determining the first orientation of the wearablecomputing device relative to the tracking device based on a differencebetween (a) the first orientation of the wearable computing devicerelative to magnetic north and (b) the first orientation of the trackingdevice relative to magnetic north.
 11. The method of claim 1, furthercomprising initiating a computing action based on the first headorientation relative to the body.
 12. The method of claim 1, wherein thedetermined first head orientation relative to the body comprisesdetermining indicates a rotation of the head relative to theforward-backward body axis.
 13. The method of claim 1, wherein thedetermined first head orientation relative to the body comprises two ormore of: (a) a rotation of the head relative to the forward-backwardbody axis, (b) a pitch of the head relative to an upward-downward bodyaxis and (c) a yaw of the head relative to the forward-backward bodyaxis and the upward-downward body axis.
 14. A non-transitory computerreadable medium having stored therein instructions that are executableto cause a computing device to perform functions comprising: detectingsensor data that is indicative of an association between movement of atracking device and body movement; in response to detecting the sensordata that is indicative of the positional association: determining aforward-backward body axis of a body associated with a wearablecomputing device; and determining a base orientation of a trackingdevice relative to the forward-backward body axis (θ_(TD-B));determining a first orientation of the wearable computing devicerelative to the tracking device (θ_(HMD-TD) _(_) ₁); and determining afirst head orientation relative to the body (θ_(H-B) _(_) ₁) based onboth: the base orientation (θ_(TD-B)) and the first orientation(θ_(HMD-TD) _(_) ₁), wherein determining the first head orientationrelative to the body (θ_(H-B) _(_) ₁) comprises offsetting the firstorientation (θ_(HMD-TD) _(_) ₁) by the base orientation of the trackingdevice relative to the forward-backward body axis (θ_(TD-B)).
 15. Acomputing system comprising: a non-transitory computer readable medium;program instructions stored on the non-transitory computer readablemedium and executable by at least one processor to: detect sensor datathat is indicative of an association between movement of a trackingdevice and body movement; in response to detecting the sensor data thatis indicative of the positional association: determine aforward-backward body axis of a body associated with a wearablecomputing device; and determine a base orientation of a tracking devicerelative to the forward-backward body axis (θ_(TD-B)); determine a firstorientation of a wearable computing device relative to the trackingdevice (θ_(HMD-TD) _(_) ₁); and determine a first head orientationrelative to the body (θ_(H-B) _(_) ₁) based on both: the baseorientation (θ_(TD-B)) and the first orientation (θ_(HMD-TD) _(_) ₁),wherein determining the first head orientation relative to the body(θ_(H-B) _(_) ₁) comprises offsetting the first orientation (θ_(HMD-TD)_(_) ₁) by the base orientation of the tracking device relative to theforward-backward body axis (θ_(TD-B)).
 16. The computing system of claim15, wherein the computing system is implemented in or takes the form ofthe wearable computing device.
 17. The computing system of claim 15,wherein the first orientation of the wearable computing device relativeto the tracking device is determined based on both: (a) magnetometerdata associated with the wearable computing device and (b) magnetometerdata associated with the tracking device.
 18. The computing system ofclaim 15, wherein the determined first head orientation relative to thebody comprises two or more of: (a) a rotation of the head relative tothe forward-backward body axis, (b) a pitch of the head relative to anupward-downward body axis and (c) a yaw of the head relative to theforward-backward body axis and the upward-downward body axis.